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Overview 


Outcomes-based funding has become an increasingly common tool for states that seek to improve 
student outcomes by explicitly linking some portion of their funding for public postsecondary 
institutions to that goal. As of 2018, twenty-five states have some form of outcomes-based funding 
(OBF) policy. Of these, twenty-two use OBF in the community college sector, and all but one of these 
states reward certificate completion. 1 


The inclusion of certificates in OBF policies has been subject to considerable scrutiny. Research on 
the impacts of OBF policies has shown significant increases in short-term certificates.2 Many have 
suggested that the surge in short-term certificates signals an unintended consequence of OBF 
policies, arguing that institutions in OBF states are being incentivized to steer students toward 
certificates with relatively lower educational and labor market value rather than toward more time- 
intensive, and presumably more valuable, degrees. 


Yet the picture is considerably more complicated. Certificates vary greatly in length, type, and labor 
market value. Additionally, states vary greatly in how they define and reward certificate 
completions in their OBF policies. States also change their OBF policies frequently. Finally, available 
databases do not adequately capture these variations. When taken together, these factors make it 
difficult to either determine or generalize about whether and how OBF policies have affected 
certificate attainment and the economic and social mobility of students. 


NOTE: For the purposes of this brief, we utilize HCM Strategists’ definition and most recent typology of 
Outcomes-Based Funding (OBF).° Broadly speaking, these policies link state appropriations for higher 
education to the accomplishment of specific student outcomes. OBF is intended to highlight the 


alignment between the state higher education funding method and the state’s higher education 
attainment and student success goals, in order to incent institutions to support these state priorities. 


1 Snyder, M. & Boelscher, S. (2018). Driving Better Outcomes: Fiscal Year 2018 State Status & Typology Update. Retrieved from HCM 
Strategist http://hcmstrategists.com/resources/driving-better-outcomes-fiscal-year-2018-state-status-typology-update 

* Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. 
Educational Evaluation and Policy Analysis, 37(4): 501-519. doi: 10.3102/0162373714560224.; Li, A. Y. & Kennedy, A. I. (2018). 
Performance funding policy effects on community college outcomes: Are short-term certificates on the rise? Community College Review, 
46(1), 3-39.; Hillman, N. W., Fryar, A. H., and Crespin-Trujillo, V. (2017). Evaluating the impacts of performance funding in Ohio and 
Tennessee. American Educational Research Journal, 55(1): 1144-170. 

3 Snyder, M. (2015). Driving Better Outcomes: Typology and Principles to Inform Outcomes-Based Funding Models. Retrieved from HCM 
Strategist http://hcmstrategists.com/drivingoutcomes/wp-content/themes/hcm/pdf/Driving%200utcomes. pdf 


This research brief lays out both the challenges and opportunities inherent in studying the effects of 
OBF on certificate attainment by: 


I. Summarizing what is known about variation in the type, length, and value of certificates; 
II. | Documenting variation in how states currently define and reward certificates in OBF 
formulas; 
III. ‘Illustrating the limitations of available datasets to capture this variation; and 
IV. Providing a set of recommendations for how the field can better examine the intended and 
unintended effects of including certificates in OBF policies. 


I. Variation in Type, Length, and Value of Certificates 


Certificates are the fastest-growing postsecondary credential nationwide. Between 2007-08 and 
2016-17, the number of certificates awarded by all degree-granting, public postsecondary 
institutions alone increased by 58 percent, compared to an increase of 49 percent in associate 
degrees and a 28 percent increase in bachelor’s degrees.*> As of 2012, one in ten American workers 
reported a certificate as their highest level of education.® 


A. Type of Certificates 


The 2006 Perkins Act characterizes certificates as either occupational, which focus on career- 
related skills and are aligned to workforce fields (e.g., computer science, education, health care); or 
academic, which are typically decontextualized from the labor market (e.g., humanities, social 
sciences). In 2016, approximately 87% of certificates were awarded in occupational fields.” 


B. Length of Certificates 


Certificates, both academic and occupational, are also classified by program length or the number of 
credit hours required for completion. States, institutions, and researchers often refer to certificates 
that can be completed in less than one year as “short-term,” while certificates that require one to 
two years are described as “medium-” or “long-term” certificates. However, “long-term” certificates 
can require as many as two to four years to complete in some states. 


Yet within these broad categories, states define certificate length differently. Figure 1 illustrates 
some of these variations in states that reward certificates in their OBF policies. For example, a 
short-term certificate in Tennessee is defined as any certificate that can be completed in less than 
one year and can require anywhere from 1 to 24 credits. In contrast, a short-term certificate in 
Indiana requires between 18 and 29 credits. This variation in how states define certificate length 


4 National Center for Education Statistics (2017). Degrees/certificates conferred by postsecondary institutions 1970-71 through 2016-17. 
Retrieved January 31, 2019 from https://nces.ed.gov/programs/digest/d18/tables/dt18_318.40.asp 

5 Miller, A., Erwin, M., Richardson, S., and Arntz, M. Collecting and Disseminating Data on Certificate Awards. (NPEC 2016). U.S. 
Department of Education. Washington, DC: National Postsecondary Education Cooperative. Retrieved March 15t 2018 from 
https://nces.ed.gov/ipeds/pdf/NPEC/data/NPEC_Paper_IPEDS_Collecting_and_Disseminating_Certificate_Awards_2016.pdf 

6 Carnevale, Anthony P., Stephen J. Rose, and Andrew R. Hanson. Certificates: Gateway to Gainful Employment and College Degrees. 
Georgetown University Center on Education and the Workforce. Washington, D.C. (2012). 

7 Authors’ calculations based on https://nces.ed.gov/programs/digest/d17/tables/dt17_320.10.asp using categorization from 
https://nces.ed.gov/surveys/ctes/tables/postsec_tax.asp. 


has significant implications for those studying the impact of OBF—a topic that is discussed 
throughout the remainder of this brief. 


Figure 1. Variation in how OBF states define short-term and long-term certificates, in credit hours 
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Notes: 

1. Seven states (about a third of states that include certificates in their OBF formulas) specify a minimum number of credit hours 
required for the certificate to qualify as an outcome metric. 

2. Tennessee defines short- and long-term certificates by credit hour requirements, but does not define a credit-hour minimum for 
short-term certificates. 

3. Ohio and Utah do not reward short-term certificates in their OBF formulas. 

4. The orange bar indicates an overlap in classification of certificates by credit hours. Texas does not classify certificates as short- 
and long-, but by levels 1, 2, and advanced. Level 1 certificates require from 15 to 42 credit hours, level 2 certificates require 
from 30 to 51, and advanced technology certificates require from 30 to 50 credit hours. 


C. Value of Certificates 


Here we focus specifically on the value of occupational certificates, which constitute the vast 
majority of certificates awarded throughout the United States each year. Occupational certificates 
are one of the main pathways into so-called “middle jobs”’—those requiring less than a bachelor’s 
degree that lead to a middle-class wage. The nation has seen a 30 percent increase in these jobs in 
the last 25 years, in contrast to the 12 percent decline in middle-class wage job opportunities 


8 Carnevale, A. P., Jayasundera T., and Hanson A.R. (2012). Career and Technical Education: Five Ways That Pay Along the Way to the B.A. 
Georgetown University Center on Education and the Workforce. Washington, D.C. Retrieved April 8° 2018 from 
https://1gyhoq479ufd3yna29x7 ubjn-wpengine.netdna-ssl.com/wp-content/uploads/2014/11/CTE.FiveWays.FullReport.pdf 


available to those with only a high school diploma.? On average, as of 2012, certificates holders 
earn 20% more than high school graduates. Bachelor’s degrees remain the best bet for economic 
opportunity; 56% of “good jobs” are those requiring at least a four year degree.1° Earnings for 
certificate-holders in certain fields, however, are comparable to those of workers with associate 
and bachelor’s degrees. For example, male workers with certificates in electronics earn more than 
65% of males with associate degrees and 48% of males with bachelor’s degrees.1! 


Certificate recipients’ labor market outcomes also vary based on race, gender, length of the 
certificate, field of study, and whether or not the recipient works in the field of the certificate.12 The 
institution granting the certificate matters as well: labor market returns to certificates earned at 
for-profit institutions tend to be lower than those earned at community colleges. 13 


Some studies have found that long-term certificates have a greater payoff in the labor market than 
short-term certificates.14 Yet other research finds that the returns on short- and long-term 
certificates vary based on gender and field. For example, Bahr et al. found that students earning a 
long-term certificate earned $2,500 to $3,600 more per year than those without a credential, with 
larger returns concentrated among men. Although women gained little when awarded a short-term 
certificate, men gained $5,200 per year, surpassing the returns of a long-term certificate.15 Several 
studies have found that returns are highest for both short- and long-term certificates in health- 
related and technical fields.1¢ 


Determining the earning effects of stackable credentials requires additional nuance. Stackable 
credentials are designed to function as sequential postsecondary completions aligned to a career 
path. There are three commonly recognized patterns of stacking. Independent stacking refers to the 


9 Carnevale, A.P., Strohl, J., Ridley N., and Artem G. (2018) Three Educational Pathways to Good Jobs: High School, Middle Skills, and 
Bachelor’s Degree. Georgetown University Center on Education and the Workforce. Washington, D.C. (2018). Retrieved November 30t 
2018 from https://cew.georgetown.edu/publications/reports/ 


10 Ibid. 


11 Carnevale, A.P., Rose S. J., and Hanson A.R (2012). Certificates: Gateway to Gainful Employment and College Degrees. Georgetown 
University Center on Education and the Workforce. Washington, D.C. Retrieved April 8 2018 from https://1gyhoq4 7 9ufd3yna29x7 ubjn- 
wpengine.netdna-ssI.com/wp-content/uploads/2014/11/Certificates.FullReport.061812. pdf; 

12 Carnevale, A.P., Rose S. J., and Hanson A.R (2012). Certificates: Gateway to Gainful Employment and College Degrees. Georgetown 
University Center on Education and the Workforce. Washington, D.C. Retrieved April 8 2018 from https://1gyhoq4 7 9ufd3yna29x7 ubjn- 
wpengine.netdna-ssI.com/wp-content/uploads/2014/11/Certificates.FullReport.061812.pdf; Bahr, P. R., Dynarski, S., Jacob, B., 
Kreisman, D., Sosa, A., & Wiederspan, M. (2015). Labor Market Returns to Community College Awards: Evidence from Michigan. A CAPSEE 
Working Paper. Center for Analysis of Postsecondary Education and Employment.; Dadgar, M., & Trimble, M. J. (2015). Labor market 
returns to sub-baccalaureate credentials: How much does a community college degree or certificate pay?. Educational Evaluation and 
Policy Analysis, 37(4), 399-418.: 399-418.; Jepsen, C., Troske, K., & Coomes, P. (2014). The labor-market returns to community college 
degrees, diplomas, and certificates. Journal of Labor Economics, 32(1), 95-121.; Xu, D., & Trimble, M. (2016). What about certificates? 
Evidence on the labor market returns to nondegree community college awards in two states. Educational Evaluation and Policy Analysis, 
38(2), 272-292.; Bailey, T., & Belfield, C. R. (2012). Community college occupational degrees: Are they worth it?. In L. Perna (Ed.), 
Preparing today’s students for tomorrow’s jobs in metropolitan America (pp. 121-148). Philadelphia, PA: University of Pennsylvania Press. 
13 Carnevale, Anthony, et al. “Certificates in Oregon - A Model for Workers to Jump Start or Reboot Careers.” CEW Georgetown, Center on 
Education and the Workforce, (2018) cew.georgetown.edu/states/oregon 

14 Dadgar, M., & Trimble, M. J. (2015). Labor market returns to sub-baccalaureate credentials: How much does a community college 
degree or certificate pay?. Educational Evaluation and Policy Analysis, 37(4), 399-418.: 399-418.; Jepsen, C., Troske, K., & Coomes, P. 
(2014). 

15 Bahr, P. R., Dynarski, S., Jacob, B., Kreisman, D., Sosa, A., & Wiederspan, M. (2015). Labor Market Returns to Community College 
Awards: Evidence from Michigan. A CAPSEE Working Paper. Center for Analysis of Postsecondary Education and Employment. 

16 Bahr, P. R., Dynarski, S., Jacob, B., Kreisman, D., Sosa, A., & Wiederspan, M. (2015). Labor Market Returns to Community College 
Awards: Evidence from Michigan. A CAPSEE Working Paper. Center for Analysis of Postsecondary Education and Employment.; Carnevale, 
A.P., Strohl, J., Ridley N., and Artem G. (2018) Three Educational Pathways to Good Jobs: High School, Middle Skills, and Bachelor’s 
Degree. Georgetown University Center on Education and the Workforce. Washington, D.C. (2018). Retrieved November 30% 2018 from 
https://cew.georgetown.edu/publications/reports/ 


accumulation of short-term credentials over time. Progression stacking is a system of short-term 
credentials articulated to a higher degree or credential. Supplemental stacking occurs when a 
student supplements a degree with a short-term credential. 


While the field recognizes the potential of progression stacking to improve completion and provide 
labor market advantages, existing research provides little evidence of impact. One study found that 
progression stacking motivated students to complete short-term certificates and pursue additional 
credentials but observed racial disparities and limited labor market gains.” Notably, scholars point 
to data limitations as obstacles to measuring the impact of stacked credentials. Specifically, both 
institutional and state data systems track the completion of discrete credentials but do not indicate 
which credentials, if any, have been stacked.18 


Taken as a whole, the considerable variation in type, length, and labor market value of 
postsecondary certificates makes it difficult to accurately generalize about the value and efficacy of 
certificates and their role in a college completion agenda.!9 Policymakers and researchers alike 
continue to debate their utility, but these debates are most useful when they include a recognition 
of the wide variation in these credentials. 


ll. | Documenting Variation in How States Include and Define 
Certificates in OBF Policy 


Twenty-one states with OBF policies currently reward community colleges for the completion of at 
least some type of certificate. While the literature offers mixed findings on the impact of OBF on 
bachelor’s and associate degrees,”° there is general consensus that OBF incentivizes institutions to 
produce more certificates—particularly short-term certificates.2! This finding is framed as an 
unintended negative effect for several reasons. First, short-term certificates are generally presumed 
to be of universally lower value in the labor market. Second, while the proliferation of short-term 
certificates may help states achieve completion goals, some intuit that the presence of certificates in 
OBF policies motivates institutions to divert students toward certificates and away from degrees, 
which can disproportionately and negatively affect underserved students.?? Additionally, some 


17 Giani, M., & Fox, H. L. (2017). Do stackable credentials reinforce stratification or promote upward mobility? An analysis of health 
professions pathways reform in a community college consortium. Journal of Vocational Education & Training, 69(1), 100-122.; Bailey, T., & 
Belfield, C. R. (2017). Stackable Credentials: Do They Have Labor Market Value? CCRC Working Paper No. 97. Community College 
Research Center, Teachers College, Columbia University. 

18 Bailey, Thomas R., and Clive Belfield. "Stackable Credentials: Awards for the Future?" (2017). New York: Community College Research 
Center. 

19 Fain, P. (2012, June 6). Not just degrees. Retrieved from https://www.insidehighered.com/news/2012/06/06/certificates-are- 
misunderstood-credentials-pay-mostly-men 

20 Tandberg, D.A., & Hillman, N.W. (2014). State Higher Education Performance Funding: Data, outcomes and policy implications. Journal 
of Education Finance, 39(3), 222-243. http://muse.jhu.edu/article/539805.; Tandberg, D. A., Hillman, N., & Barakat, M. (2014). State 
Higher Education Performance Funding for Community Colleges: Diverse Effects and Policy Implications. Teachers College Record, 
116(12). 

21 Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. 
Educational Evaluation and Policy Analysis, 37(4): 501-519. doi: 10.3102/0162373714560224.; Li, A. Y. & Kennedy, A. I. (2018). 
Performance funding policy effects on community college outcomes: Are short-term certificates on the rise? Community College Review, 
46(1), 3-39.; Hillman, N. W., Fryar, A. H., and Crespin-Trujillo, V. (2017). Evaluating the impacts of performance funding in Ohio and 
Tennessee. American Educational Research Journal, 55(1): 1144-170. 

22 Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. 
Educational Evaluation and Policy Analysis, 37(4): 501-519. doi: 10.3102/0162373714560224. 


researchers suggest that institutions are creating certificates of little value in order to be rewarded 
for them under their state formulas. 


Yet discerning the relationship between OBF, certificate production, and students’ long-term 
outcomes (i.e., educational attainment and living wage employment) is more complicated. Not only 
do certificates vary along a number of dimensions as noted above; but state OBF policies also vary 
in terms of which certificates are rewarded and under what conditions. 


This variation in how states define the certificates rewarded under their formulas has not been 
captured in the preeminent literature on OBF.?3 Typically, OBF policies are categorized as either 
rewarding certificates or not. Yet because certificates vary so much in terms of length, type, and 
labor market value, analysis that does not consider which certificates are being rewarded through 
OBF may be over-generalizing about the effects of OBF on certification production. In addition, 
because existing research does not account for differences in how OBF polices reward certificates, it 
cannot accurately inform the field about whether including certificates in OBF policy is an effective 
strategy for increasing educational attainment and living-wage employment. Nor can it distinguish 
which certificate-rewarding strategies are most effective in either increasing credential attainment 
or improving the quality of the workforce. 


It is also important to recognize that states frequently make changes to their formulas. While formal 
review of state formulas is often aligned to legislative sessions, informal changes to formulas can 
happen more frequently.2425 For example, New Mexico changed its funding formula annually for the 
first five years of implementation.2¢ These changes can in some cases have substantial impacts on 
the ways in which certificates are treated. 


Below, we provide a two-step process for accurately determining whether and how states include 
certificates in their OBF policies. 


Step 1. Determine which states have OBF policies and whether policies include 
certificates 


It can be difficult to accurately determine which states have OBF at specific points in time and, if so, 
what type. This exercise becomes more complicated when examining the effect of OBF on certificate 
production, given the variability in certificates documented above. Additionally, there is a third 
layer of complexity: the variation in how states define and categorize certificates in OBF policy. 


Table 1 provides a 50-state analysis indicating which states have OBF in place, which OBF policies 
reward certificates, and, where applicable, which types of certificates are rewarded. To construct 


23 Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. 
Educational Evaluation and Policy Analysis, 37(4): 501-519. doi: 10.3102/0162373714560224.; Li, A. Y. & Kennedy, A. I. (2018). 
Performance funding policy effects on community college outcomes: Are short-term certificates on the rise? Community College Review, 
46(1), 3-39.; Hillman, N. W., Fryar, A. H., and Crespin-Trujillo, V. (2017). Evaluating the impacts of performance funding in Ohio and 
Tennessee. American Educational Research Journal, 55(1): 1144-170. 

*4Snyder, M. & Boelscher, S. (2018). Driving Better Outcomes: Fiscal Year 2018 State Status & Typology Update. Retrieved from HCM 
Strategists http://hcmstrategists.com/resources/driving-better-outcomes-fiscal-year-2018-state-status-typology-update 

25 Research for Action. (2018). Outcomes-based Funding Equity Toolkit. Retrieved from Research for Action 
httpes://www.obfequitytoolkit.org/ 

26 Ibid. 


this table we reviewed information on the types of certificates rewarded in OBF policies in publicly 
available policy documents, OBF formula definitions, and legislative documents (See Appendix A). 
In cases of discrepancy or ambiguity, we consulted state policymakers for verification. 


Table 1. 50-state overview of outcomes-based funding (OBF) policies: OBF policies that include two-year 
sector institutions, reward certificates, and reward specific types of certificates as of FY2018 


OBF explicitly OBF does 


OBF rewards OBF rewards 


State OBF2’ as OBF in two- | OBF rewards rewards both __ not specify 
of FY2018 yearsector | certificate types of 
completion only only and | D certificates 
certificates rewarded 
AZ 
AR e e e e 
CA 
CO e @ e e 
CT 
DE 
FL e e e e 
GA 
HI e e e e 
ID 
IL e e e e 
IN e e e e 
IA 
KS 
KY ® e e e 
LA e e e e 
ME e 
MD 
MA 
MI e e e e 
MN 
MO 
MT e e e e 


27 Snyder, M. & Boelscher, S. (2018). Driving Better Outcomes: Fiscal Year 2018 State Status & Typology Update. Retrieved from HCM 
Strategists http://hcmstrategists.com/resources/driving-better-outcomes-fiscal-year-2018-state-status-typology-update 


OBF explicitly OBF does 


OBF rewards OBF rewards 


State OBF2’ as OBF in two- | OBF rewards rewards both __ not specify 
of FY2018 yearsector | certificate types of 
completion only only and | D certificates 
certificates VEC 
NH 
NJ 
NM e e e e 
NY e e e ° 
NC e e e e 
ND e e 
OH e e e e 
OK 
OR e 
PA e 
RI 
SC 
SD 
T™ e e e e 
TX e e e e 
UT e e e e 
VT 
VA e e e e 
WA e e e e 
WV 
Wi e e e e 
TOTAL 25 22 21 6) 2 9 10 


Notably, of the 21 states that reward institutions for certificate completion via OBF policies: 


e Nostate exclusively rewards short-term certificates; 

e Two states reward only long-term certificates; 

e Nine states explicitly reward some type of both short- and long-term certificates; and 

e Ten have adopted OBF policies that reward certificates but do not specify which types of 
certificates are rewarded. 


Step 2. Identify how states reward certificates in OBF policy 


When states do reward institutions for certificate completion in OBF policy, their approaches to 
doing so vary considerably. We reviewed documentation from state websites and media outlets in 
2018 for each of the 21 states that currently reward some type of certificate in their two-year OBF 
policies. To confirm our characterization of how certificates were rewarded under state OBF 
policies, we triangulated the collected documentation with data reports and consulted with state 
policymakers in cases where there were discrepancies or ambiguity in the policy documentation. 


Table 2 summarizes the conditions under which certificates are rewarded in these policies. 


Table 2. Criteria for rewarding certificates in outcomes-based funding (OBF) policies: Limiting and 
incentivizing factors across 21 state policies in 2018 


LIMITING FACTORS INCENTIVING FACTORS 


HIGH- BONUS 
WEIGHTED | MINIMUM SEMAND LIMITS BONUS FOR rontiGie 


STATES 
LESS THAN | CREDIT HOUR FIELDS NUMBER FOCUS DEMAND 
DEGREES REQUIREMENT ONLY REWARDED POPULATION@ FIELD» 
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a. States in this category reward all certificates meeting their minimum credit hour requirements, but the way bonuses are applied 
varies: we include states which reward bonus points for each certificate, states that set bonus weights for focus population, and 
states that include focus population metrics in their OBF formula. Focus populations are student groups that states identify as 
priority subgroups, e.g., low-income students, underprepared students, or underrepresented minorities. 

b. States in this category reward all certificates meeting the minimum requirements and apply bonuses to certificates in certain 
high-demand fields. In contrast, states identified in the fifth column only reward certificates in the fields identified by the state. 

c. Colorado only rewards short-term certificates (those that require less than a year to complete; typically 24 credit hours) that 
meet the federal “gainful employment” definition. 

d. Colorado’s formula only rewards one short-term certificate (those that require less than a year to complete; typically 24 credit 
hours)within the reporting period, but does not limit the number of longer-term certificates or degrees rewarded under the 
formula. 

e.  Florida’s formula only rewards certain categories of certificates, e.g., Career and Technical Certificate, Advanced Technical 
Certificate. See details here: https://www.floridacollegesystem.com/sites/www/Uploads/CompletionRate_1718 _Model.pdf. 

f. Montana applies weights on credentials by institutional mission; completions across metrics are weighted at 30% at flagship 
institutions, 40% at four-year regionals, and 30% at two-year colleges. 

g. | Montana only includes one-to-two-year certificates that are recognized as Certificates of Applied Science and less-than-one-year 
certificates that are Certificates of Technical Studies in their OBF formula in the Montana University System. 

h. | Nevada rewards all one-to-two-year certificates that require 30 or more credit hours and awards bonuses to all certificates in 
STEMH fields, but only rewards less-than-one-year certificates that prepare students to take state-, national-, and/or industry- 
recognized certification or licensing examinations. 

i. New Mexico only rewards less-than-one-year certificates in STEMH fields, but one-to-two-year certificates are rewarded 
regardless of field. 

j. New York’s formula only applies to SUNY and CUNY systems. No information on limiting factors is publicly available. 

k. — North Carolina rewards one point to each student who earns a credential at any level, including a diploma, certificate, or 
associate degree. Multiple credentials earned will only be counted as one toward the metric of curriculum completion. See 
details here: https://www.nccommunitycolleges.edu/sites/default/files/data- 
warehouse/2018 performance measures report_071118 final_1.pdf#overlay-context=analytics/state-and-federal- 
performance-measures (p.12). 

I. Ohio’s policy will reward more than one credential in an academic year, but while the first is counted at 100 percent, 
subsequent credentials are rewarded at 50 percent. 

m. Tennessee rewards only less-than-one-year certificates that are defined as technical certificates. 

n. | Washington only rewards one point for completion per student every academic year. See details here: 
https://www.sbctc.edu/resources/documents/about/agency/initiatives-projects/sai-3-0-criteria-and-documentation-5-16- 
18.pdf. 


The six columns in Table 2 fall into two categories. The first four columns are ways in which states 
are limiting the number of certificates rewarded under OBF and include: 


e Weighted Less Than Degrees: Eight states reward certificates, but weight (i.e., reward) them 
less than degrees. Certificate weights vary from .1 to .67. 

e Credit Hour Minimum: In seven states, certificates must meet a credit hour minimum to be 
rewarded under OBF. Typically, a less-than-one-year certificate requires a credit hour 
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minimum between 15 and 29 credit hours, while a one-to-two-year certificate requires 
minimums between 20 and 30 credit hours. 

e High-Demand Fields Only: Seven states reward specific prioritized fields, usually in STEM 
and Health, that are aligned with state workforce development priorities. Only certificates 
earned in these state-defined high-demand fields are counted in the formula. 

e Limitations on Number of Certificates Rewarded: Seven states limit the number of 
certificates rewarded by their formula if students earn more than one credential in that 
reporting period. Typically, if students earn multiple certificates, states in this category only 
reward one certificate. If a student earns a certificate and an associate degree in the same 
reporting period, the certificate would not be rewarded or would be discounted. 


The last two columns describe conditions that may provide an incentive for institutions to produce 
certificates: 


e Bonus for Focus Population: Nineteen states award a bonus (i.e., extra weight or premium) 
when a certificate is earned by a student from a focus population, such as low-income 
students. 

e Bonus for High-Demand Fields: Nine states provide a bonus when institutions award 
eligible certificates in high-demand fields, such as STEM. Typically, high-demand fields are 
aligned with state workforce needs. Certificates earned in high-demand fields in these states 
are rewarded with either more weight or as a separate outcome than other certificates in 
OBF formulas. 


Specific notable patterns seen in Table 2 include the following: 


States vary substantially in the number of criteria placed on certificates rewarded by their 
OBF formulas, and in some cases have criteria that both limit and incentivize certificate 
production. While states such as Hawaii, Illinois, Kentucky, Michigan, and New York place no 
restrictions on certificates in their outcomes-based funding policies, other states have a mix of 
conditions that in some cases restrict and in other cases incentivize certificates under the formula. 
For example, Nevada and Utah weight certificates less than degrees and require a minimum 
number of credit hours. But they also provide bonuses for certificates in high-demand fields and for 
those earned by students in focus populations. 


Eight states weight certificates less than degrees, while the remaining 13 states reward 
certificates and degrees equally. For example, Arkansas weights certificates so that they are 
rewarded at half the rate of degrees, while Colorado weights certificates so that they are rewarded 
at a quarter the rate of degrees. In contrast, degrees and certificates both count equally as 
completions in 13 state formulas. 


Seven states include minimum credit hour requirements for certificates to be rewarded in 
the formula. Ohio and Utah only reward certificates of 30 credits or more. For Indiana, the 
minimum is 18 credits; for Montana, 16, for Nevada, 9, for Texas, 15, and for Washington, 20. 


Seven states only reward certificates in specific fields of study identified by the state higher 
education authority. Florida only rewards certificates earned from advanced technical programs 
or certain career and technical programs based on economic development priorities. New Mexico, 
Nevada, and Tennessee reward all long-term certificates but require that short-term certificates be 
either in a STEM or healthcare field, or in a state-recognized technical field, to count under the 
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formula. Colorado only rewards certificates of less than one year that meet the federal “gainful 
employment” definition. Montana only rewards Certificates of Applied Science or Certificates of 
Technical Studies. Wisconsin only rewards certificates that correspond to one of the state’s most in- 
demand occupations. 


Seven states do not fully reward multiple certificates earned in the same reporting year. For 
example, in New Mexico, Texas, and Virginia, if a student earns a certificate and an associate degree 
in the same year, only the associate degree is rewarded by the OBF formula. In six out of the seven 
states, if a student earns two certificates in the same year, only one of the two is rewarded. Ohio is 
an exception; if a student earns two certificates in the same year both count, but the second 
certificate is discounted by 50 percent. 


Nearly all states (19 out of 21) award bonuses for certificates awarded to underserved or 
focus populations. Focus populations can include low-income students, underprepared students, 
students of color, adults, and veterans. Only North Carolina and Michigan do not reward institutions 
with additional funding for the successful attainment of a certificate by students from focus 
populations. 


Nine of 21 states award bonuses for certificates in a high-demand field. High-demand fields 
are typically defined as STEM fields and health care. States award bonuses for these certificates in 
different ways: some apply a bonus point for each certificate earned in high-demand field, some 
include a metric for high-demand field certificate production in their formulas, while others use 
bonus weights on these certificates to reflect the value of high-demand industries. 


The considerable variation detailed in Tables 1 and 2 does not support the argument that any 
inclusion of certificates in OBF policies incentivizes the production of short-term certificates at the 
expense of other credentials. While this may well be true in some states, a close examination of the 
variation in certificates, and in OBF policies themselves, paints a far more complex picture. 


Ill. Determining the Degree to Which Data Sources Accurately 
Capture State Variation in Certificates Rewarded under OBF 
Policies 


Existing research on the impact of OBF on certificate production relies heavily, if not exclusively, on 
data available in the Integrated Postsecondary Education Data System (IPEDS). However, IPEDS 
does not collect data on several key characteristics of postsecondary certificates or OBF policy 
elements such as whether a certificate is the highest credential earned in an academic year, 
minimum credit hour requirements for certificates, or student characteristics such as residency 
status, age, and socioeconomic status. These limitations hamper the capacity of researchers who 
use IPEDS to conduct analyses that accurately capture variation in the impact of OBF on certificate 
production and their value. 


In contrast, statewide longitudinal data systems often include more nuanced information about 
certificates and the students receiving them, as well as specific OBF policy elements. Yet these 
administrative datasets also present limitations for rigorous research, particularly with regard to 
variation in quality and consistency across states. The strengths and limitations of each dataset for 
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studying the effects of OBF on certificate production are described below. Additionally, we describe 
how workforce data may strengthen research on certificate production. 


A. The Integrated Postsecondary Education Data System (IPEDS) 


IPEDS is a system of surveys conducted annually by the National Center for Education Statistics 
(NCES). IPEDS is the most commonly used data source for measuring the impact of OBF. As of 
December 2018, eight peer-reviewed journal articles include analyses of the impact of OBF policies 
on certificate and degree production in community or technical colleges. Of these, seven utilize 
IPEDS as their primary data source. In addition, the two articles that specifically examine certificate 
production in OBF states both use IPEDS.28 IPEDS includes longitudinal data, collected annually 
since 1980, from all postsecondary institutions (i-e., colleges, universities, technical and vocational 
institutions) that participate in federal student financial aid programs. 


Benefits of Using IPEDS to Examine Certificates in OBF 


e The comprehensive coverage of U.S. postsecondary institutions allows for comparative 
analyses at multiple levels, including state, sector, and institution levels. 

e Longitudinal data collection enables long-term trend comparisons for a range of 
institutional characteristics and outcomes that have not experienced significant reporting 
changes over time. 

e ©The inclusion of total number of certificates awarded by major fields and selected 
student populations allows for some tracking of changes in certificate production in OBF 
states and comparison of certificate production among OBF and non-OBF states. 

e Because IPEDS is publicly available and comprehensively documented it is widely used 
and analyses can be replicated if researchers fully display their sampling and methods. 


Limitations of Using IPEDS to Examine the Impact of OBF on Certificate Production”? 


IPEDS has several significant weaknesses that limit its utility for examining the impact of rewarding 
certificate completion in OBF policies. Most notably, IPEDS surveys require institutions to report on 
a common set of metrics that do not capture the variations in certificates themselves, in how states 


28 Tandberg, D. A., Hillman, N., & Barakat, M. (2014). State Higher Education Performance Funding for Community Colleges: Diverse 
Effects and Policy Implications. Teachers College Record, 116(12), n12.; Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). 
Evaluating the impacts of “new” performance funding in higher education. Educational Evaluation and Policy Analysis, 37(4): 501-519. 
doi: 10.3102/0162373714560224.; Kelchen, R. & Stedrak, L. J. (2016). Does Performance-Based Funding Affect Colleges’ Financial 
Priorities?. Journal of Education Finance, 41(3), 302-321.; Hillman, N. W., Fryar, A. H., and Crespin-Trujillo, V. (2017). Evaluating the 
impacts of performance funding in Ohio and Tennessee. American Educational Research Journal, 55(1): 1144-170.; McKinney, L., & 
Hagedorn, L. S. (2017). Performance-based funding for community colleges: Are colleges disadvantaged by serving the most 
disadvantaged students?. The Journal of Higher Education, 88(2), 159-182.; Hu, X., & Villarreal, P. (2018). Public tuition on the rise: 
Estimating the effects of Louisiana’s performance-based funding policy on institutional tuition levels. Research in Higher Education, 1-34.; 
Li, A. Y., Gandara, D., & Assalone, A. (2018). Equity or disparity: Do performance funding policies disadvantage 2-year minority-serving 
institutions? Community College Review, 46(3), 288-315.; Li, A. Y. & Kennedy, A. I. (2018). Performance funding policy effects on 
community college outcomes: Are short-term certificates on the rise? Community College Review, 46(1), 3-39; Li, Amy Y., Kennedy, Alec I., 
& Sebastian, Margaret L. (2018). Policy Design Matters: The Impact of Performance Funding Policies on Credential Completion at 
Community Colleges (WISCAPE POLICY BRIEF). Madison, WI: University of Wisconsin-Madison, Wisconsin Center for the Advancement of 
Postsecondary Education (WISCAPE). 

22 The IPEDS surveys we use for this analysis include Institution Characteristics and Completions. We refer to the combined dataset of 
these two surveys as ‘IPEDS data’ in this brief. 
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define them, or in how they reward them under OBF. Specific areas of this disconnect are as 
follows: 


IPEDS classifications of short and long-term certificates do not align with OBF policy 
classifications. As noted in Table 2, seven of 21 OBF states that reward certificates set credit hour 
minimums for certificates to be eligible for funding through the formula. However, IPEDS does not 
specify a credit hour minimum for institutions when reporting certificates for the “less than one 
academic year” category. 


Table 3 lists how IPEDS instructs institutions to report different categories of certificates. According 
to IPEDS, certificates classified as short-term, or “less than one year,” include a wide range of 
certificates, from those requiring as few as 3 credits to certificates requiring 29 credits (NPEC, 
2012). Additionally, IPEDS makes it optional for institutions to report certificates earned with 
fewer than 12 credits that are approved at the institution or regional level, but many do report 
these data in their IPEDS surveys.*° 


Table 3. Reporting categories of certificates in IPEDS 


CATEGORY OF 


CERTIFICATE | IPEDS REPORTING INSTRUCTION | NOTE 


Less than 900 clock hours; less 
than 30 semester credit hours; or 
less than 45 quarter credit hours. 


At least 900 but less than 1800 
clock hours; at least 30 but less 
than 60 semester credit hours; or at 
least 45 but less than 90 quarter 
credit hours. 


Also referred to as “short-term 
certificates” 


Less than one academic 
year 


At least one but less than 
two academic years 


Also referred to as “medium-” or 
“moderate-term certificates” 


1800 or more clock hours; 60 or 
more semester credit hours; or 90 
or more quarter credit hours. 


Figure 2 compares state definitions of short- and long-term certificates with the definitions used by 
IPEDS. It includes the seven states that set credit hour minimums for short- and long-term 
certificates to be rewarded under their OBF formulas, as well as Tennessee, which does not specify 
a credit hour minimum, but sets 24 credit hours as the threshold between short- and long-term 
certificates. The figure clearly demonstrates the ways in which IPEDS definitions do not align with 
state definitions of certificate length. 


At least two but less than 
four academic years 


Also referred to as “long-term 
certificates” 


Figure 2. Comparing state definitions for short- and long-term certificates with those used in IPEDS 


3° Ibid. 
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Notes: 

1. Seven states (about a third of states that include certificates in their OBF formulas) specify a minimum number of credit hours 
required for the certificate to qualify as an outcome metric. 

2. Tennessee defines short- and long-term certificates by credit hour requirements, but does not define a credit-hour minimum for 
short-term certificates. 

3. Ohio and Utah do not reward short-term certificates in their OBF formulas. 

4. The orange bar indicates an overlap in classification of certificates by credit hours. Texas does not classify certificates as short- 
and long-, but by levels 1, 2, and advanced. Level 1 certificates require from 15 to 42 credit hours, level 2 certificates require 
from 30 to 51, and advanced technology certificates require from 30 to 50 credit hours. 


Figure 2 indicates that research based on IPEDS classifications would over-count qualifying short- 
term certificates in each of these OBF states, and mis-specify some certificates that count as “long- 
term” in Tennessee as short-term. For these reasons, analyses based exclusively on IPEDS cannot 
accurately distinguish between definitions of short- and long-term certificates for all OBF states. 


IPEDS only includes total certificates awarded by institution. But some OBF formulas do not 
recognize and/or discount multiple certificates earned within one reporting year by the 
same student. IPEDS reports total certificates awarded by an institution for each academic year. In 
contrast, some OBF policies only reward one credential per field per student per academic year. For 
example, in New Mexico, if a student earns two certificates in one year on the way to an associate 
degree within that academic year, institutions can report three credentials to IPEDS. However, the 
formula only recognizes the associate degree earned. This discrepancy between what is reported to 
IPEDS and what states recognize in their OBF formulas poses challenges for those seeking to 
understand the degree to which the number of certificates awarded in a state is due to OBF. 


IPEDS does not include consistent data on student subgroups typically included in OBF 
formulas. Certain student characteristics were not available in IPEDS until after the 
implementation of some states’ OBF formulas, e.g., age categories and Pell-recipient status. As a 
result, changes in outcomes of these student populations cannot be measured—even though 
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several states provide incentives for institutions to prioritize these students in their OBF policies.31 
While NCES continues to expand postsecondary data elements and improve data reporting 
procedures across IPEDS surveys, the range of student groups identified as focus populations in 
OBF formulas is not fully available in the database. As a result, analyses cannot assess how 
certificate attainment across these student groups has changed since OBF formulas have been 
implemented, nor whether incentives to produce certificates among these subgroups has had an 
effect. 


B. Statewide Longitudinal Data Systems (SLDS) 


SLDS provide state-specific, student-level longitudinal data that may include a range of pre-K, K-12, 
postsecondary, and workforce variables. 


The scope and quality of SLDS vary considerably across states. States with higher data capacity and 
well-developed SLDS link pre-K through workforce student record data. Many SLDS that track 
postsecondary education support analyses that can be used to examine a wide array of student 
behaviors and outcomes, including student mobility, field of study, transfer between two-and four- 
year sectors, and post-baccalaureate employment, but others are more limited. 32 


Benefits of Using SLDS to Examine Certificates in OBF 


SLDS can capture the specific types of certificates rewarded by OBF in a particular state. 
SLDS frequently include more detail about certificates than does IPEDS. Specifically, while IPEDS 
uses a minimum of 30 credit hours to distinguish long-term certificates, states frequently use 
different credit hour minimums in determining which certificates to categorize and reward 
certificate completion. Utilizing a state’s longitudinal data system can provide data elements that 
capture this type of variation. 


Example: Tennessee 


Tennessee defines short-term certificates as those requiring less than 24 credit hours. IPEDS 
classifies any certificate with less than 30 credit hours as a short-term certificate. However, 
Tennessee’s SLDS can be used to construct a count of short-term certificate production that more 
closely aligns with how they are being incentivized through the OBF policy. As can be seen in Table 
4, IPEDS over-counts the number of short-term certificates rewarded by Tennessee’s OBF formula 
by 19 percent. 


In contrast, IPEDS undercounts the number of long-term certificates rewarded by Tennessee’s OBF 
formula. As Table 4 indicates, Tennessee defines long-term certificates as “at least 24 credit hours 


31 For example, degree/credential attainment by adult students only became measurable in 2012 with the addition of the student age 
group variable. In IL, MN, and TN, states provide bonus weights on certificates earned by adult students. 

32 Levesque, K., Fitzgerald, R., and Pfeiffer, J. (2015). A Guide to Using State Longitudinal Data for Applied Research. [Analytic Technical 
Assistance and Development by Institute of Education Sciences] 
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completed.” IPEDS defines long-term certificates as “more than 30 credit hours completed.” As a 
result, IPEDS identifies 23 percent fewer long-term certificates than does Tennessee’s SLDS. 


Table 4. Reported number of certificates awarded in Tennessee by data systems (2016) 


CERTIFICATE TYPE | SLDS | IPEDS | % DIFFERENCE | EXPLANATION FOR MISALIGNMENT 
Credit hours required: IPEDS defines as 
Shorten “less than 30 credit hours” but Tennessee 
es 3345 4033 19% uses 23 as the maximum. The state’s 
certificates 


SLDS allows for analyses that accurately 
capture this nuance. 


Numbers of credit hours required: IPEDS 
defines as “30 or more credit hours” but 
2302 1818 23% Tennessee uses 24 as the minimum. The 
state’s SLDS allows for analyses that 
accurately capture this nuance. 


Long-term 
certificates 


Note: Tennessee’s SLDS provides data on credits earned for credentials at the student level and allows researchers to classify types of 
certificates to align with Tennessee’s formula. 

SLDS contains individual student-level characteristics that researchers can use to track 
changes incentivized by some OBF formulas. As noted above, OBF policies vary in terms of 
whether and how they recognize certificate attainment among specific student populations. 
Frequently, IPEDS does not capture these student characteristics. 


Example: Indiana 


Indiana only rewards credentials earned by state residents within its OBF formula. However, IPEDS 
differentiates credential attainment by U.S. citizenship only. As a result, IPEDS over-reports the 
number of certificates recognized by Indiana’s OBF policy by 11 percent, as seen in Table 5. 


Table 5. Number of certificates awarded in Indiana by citizenship vs. residency status (2016) 


IPEDS: 30 OR SLDS: 30 OR MORE 
CERTIFICATE MORE CREDIT CREDIT HOURS % EXPLANATION FOR 


TYPE HOURS (ALL US (INDIANA RESIDENTS | DIFFERENCE MISALIGNMENT 
(e745 5) ONLY) 


Indiana only rewards 
certificates earned by state 
Long-term p residents but IPEDS provides 
certificates ene 12° aie the number of certificates 
earned by US citizens 
regardless of residency. 


Note: Indiana added short-term certificates as outcomes in its 2017-19 updates of metrics, but the 2017 SLDS data were not available at 
the time of this analysis. 


Limitations of Using SLDS to Examine Certificates in OBF 


SLDS can support an array of research methods that examine the effect of OBF policies on 
certificates within states, but they are also limited in the following ways: 
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e SLDS are not publicly available, and data files may require significant preparation to 
support analysis. Although many states have well-established processes for making SLDS 
available to researchers, data sharing agreements, timelines, and costs vary by state. In 
addition, SLDS are designed for administrative purposes and SLDS data files often require 
extensive verification and cleaning to be completed prior to analysis. 

e Although most states developed SLDS around the same time, the data collection cycle 
varies by state. For example, some states collect degree attainment once per academic 
year, but others collect data every semester.?3 

e The quality and availability of data varies by state. We also observe variations in data 
reporting practices and availability of data before and after the implementation of OBF. 
Some states only began tracking OBF formula outcomes in their SLDS after the policy was 
adopted. The absence of pre-OBF data for some outcomes makes it difficult to measure the 
effects of the policy. In addition, inconsistency in the availability of some data items across 
states limits the feasibility of extensive cross-state analyses. 

e Notall SLDS connect postsecondary and workforce data. This lack of consistency across 
different SLDS systems hampers researchers’ ability to track the labor market impact of 
certificates in OBF formulas that reward certificate completion. 


C. State Workforce Data 


State workforce data are typically collected annually at the individual level by Departments of 
Labor and Workforce Development in each state. They capture employment after graduation, 
median wage, education level required by different occupations and industries, and employment 
aligned with state priorities. 


When combined with postsecondary data, workforce data can support a better 
understanding of credential attainment in prioritized fields and industries rewarded by 
OBF. Nine of 21 states prioritize certificates in specific fields or industries in their OBF policies. 
These industries are identified through state departments of workforce as areas of growth or need, 
and credentials aligned to those industries are rewarded through OBF. Workforce data are 
therefore critical in understanding whether this prioritization has increased aligned credential 
production. 


Both IPEDS and SLDS data enable researchers to examine credential production by Classification of 
Instructional Programs (CIP) codes, but not all states use CIP or major fields to designate the 
prioritized fields that will be rewarded through OBF. For example, Louisiana and Wisconsin 
periodically identify priority jobs using Standard Occupational Classification (SOC) codes which are 
not directly available in either IPEDS or SLDS data. However, a crosswalk between workforce SOC 
and CIP codes exists, which enables researchers to obtain an accurate number of credentials earned 
in prioritized fields. Research aimed at understanding the impact of prioritizing workforce-aligned 
credentials in OBF formulas must take this additional step to ensure that they are measuring these 
outputs accurately. 


33 Levesque, K., Fitzgerald, R., and Pfeiffer, J. (2015). A Guide to Using State Longitudinal Data for Applied Research. [Analytic Technical 
Assistance and Development by Institute of Education Sciences] 
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Alternatively, if researchers fail to attend to the presence of misalignment between SOC and CIP 
classifications, the discrepancies can substantially skew an analysis of OBF impact. We present the 
following example of certificate production in Louisiana to highlight the differences in results. 


Example: Louisiana 


In Louisiana’s current OBF policy, workforce certificates that lead to “four- and five-star jobs” earn 
a bonus under the OBF formula.34 Combining IPEDS data with data from Louisiana’s Workforce 
Commission? in Figure 3, we present a summary of certificate production from 2010 to 2016, 
highlighting changes since 2014, the year that OBF was adopted in Louisiana. 


Figure 3. Certificate production in Louisiana (Data Source: IPEDS 2010-2016) 
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Note: The red line indicates the year Louisiana adopted performance based funding (2014). 


Figure 3 shows that the overall number of certificates reported in IPEDS declined by 18 percent, as 
did the number of certificates not prioritized in the state’s OBF formula. Yet the number of 
prioritized certificates increased by 8 percent (see Appendix B for more details). If an analysis of 
the effect of OBF on certificate production does not capture these distinctions, it could lead to very 
different conclusions about the effects of OBF. 


Similarly, in a recent study of the impact of OBF with STEM incentives on STEM bachelor’s degree 
completions, Li used workforce data to account for variation in prioritized outcomes and found that 
STEM incentives resulted in increased production of STEM bachelor’s degrees.3¢ Analogous 


34 Retrieved from https://regents.la.gov/wp-content/uploads/2018/03/4c-Outcomes-Based-Funding-Formula-Overview.pdf 
35 Retrieved from http://www.laworks.net/Stars/default.aspx 

36 Li, A. Y. (2018). Performance funding policy impacts on STEM degree attainment. Educational Policy. Retrieved from 
http://journals.sagepub.com/doi/abs/ 10.117 7/0895904818755455. 


19 


research incorporating workforce data is needed to assess the effects of the prioritization of fields 
through OBF on certificate production in the two-year sector. 


IV. Implications for Research on OBF and Certificates 


Research on OBF and certificates is complicated by four factors. First, there is significant and wide- 
ranging variation in certificate definitions, length, type, and labor market value across states. 
Second, whether and how states include certificates in their OBF policies varies widely as well. 
Third, state formulas change often, which can have implications for which certificates are rewarded 
and how. Lastly existing databases, particularly IPEDS, do not capture these variations. Here, we 
highlight a range of recommendations for researchers to consider when examining certificate 
production following the implementation of OBF policies. 


Analysis of OBF’s impact on any outcome, and especially certificates, requires accuracy in 
identifying the specific outcomes rewarded under each state’s funding formula. For example, 
one study examining the impact of OBF on short-term certificates included Ohio, whose formula 
does not reward short-term certificates; and another used IPEDS to measure short-term certificate 
production in a state whose credit hour definition for short-term certificates is misaligned with 
those of IPEDS.3738 Such data limitations in IPEDS prevent studies from accurately accounting for 
the type of certificates some states award.39 


Similarly, seven out of the 21 states examined in this brief only reward certificates if they are ina 
field that aligns with state workforce development or in STEM and health fields. For example, 
Wisconsin only rewards degrees and certificates in the top 50 occupations identified by the 
Department of Workforce Development as projected new or opening fields. In 2016, only 5 percent 
of the total number of certificates reported by IPEDS were earned in major fields on the top 50 
occupation lists.4° Future research on the impact of OBF in these states would be more accurate if it 
isolated the impact of the policy on certificates in the fields of study directly rewarded by the states’ 
formula. 


Researchers should confirm with states how they define and reward certificates in OBF 
policy before examining the impact of the policy, and clearly indicate which certificates they 
include in their analyses. States do not always make public their rules for defining, rewarding, or 
tracking certificate production in OBF policies. Moreover these rules can change frequently, which 
in turn can lead to changes in data reporting. For example, Nevada did not include short-term 
certificates in its OBF policy until 2013, hence some institutions only tracked short-term certificates 
after the policy update. To address such issues, it is often necessary to talk with one or more state 
policymakers to obtain a complete and accurate description of the treatment of certificates in OBF 


37 Li, A. Y. & Kennedy, A. |. (2018). Performance funding policy effects on community college outcomes: Are short-term certificates on the 
rise? Community College Review, 46(1), 3-39. 

38 Hillman, N. W., Tandberg, D. A., and Fryar, A. H., (2015). Evaluating the impacts of “new” performance funding in higher education. 
Educational Evaluation and Policy Analysis, 37(4): 501-519. doi: 10.3102/0162373714560224. 

39 Sykes, A. (2012). Defining and Reporting Subbaccalaureate Certificates in IPEDS (NPEC 2012-835). U.S. Department of Education. 
Washington, DC: National Postsecondary Education Cooperative. Retrieved March 15% 2018] from 
https://nces.ed.gov/pubs2012/2012835.pdf 

40 See Appendix B for methodology. 
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policy. While this process can be time-consuming, it is foundational to an accurate analysis of how 
OBF policies have affected certificate production. 


Future research should examine whether OBF policies have a larger impact on certificates in 
high-demand fields than on other certificates. Sixteen states explicitly incentivize institutions to 
focus on producing more certificates that are aligned with state workforce and economic 
development: seven states use high-demand fields to limit what certificates are included as 
outcomes, while nine use incentives such as more weights or premiums. When examining the 
policies’ effects on certificate production, researchers should consider exploring how production 
varies across fields. 


An analysis of the impact of OBF on certificate production should explore effects on student 
attainment across focus populations. IPEDS reports certificate completions disaggregated by 
certain demographic characteristics. This allows researchers to examine if OBF policies with 
bonuses for some focus populations result in more certificates for those populations than OBF 
policies without a bonus. For example, Montana provides bonuses to institutions for certificates 
earned by adult and American Indian students. IPEDS completion data indicates that in 2015, 1,225 
Montana students graduated with a certificate and among them, 56 percent were adult students 
and 16 percent were American Indian students. Further research should examine how such 
bonuses affect certificate attainment for targeted subgroups as compared to states without such 
bonuses. 


Research should examine if the proliferation of certificates under OBF differs between states 
that reward certificate and degree production equally and those that weight certificates 
lower. Some researchers have hypothesized that community colleges respond to OBF by producing 
more certificates and fewer associate degrees. Yet some states don’t reward certificate production 
at all in their policies, and others reward less for certificates than they do for degrees (see Table 
2).41 Future research should provide a more nuanced exploration of the effects of OBF on certificate 
production by taking into account these kinds of variations. The incentive for institutions to “shift” 
from associate degree production to certificate production is likely to vary at least to some degree 
according to these differential weights. 


Linking data sources such as SLDS and state workforce data can help address challenges 
posed by the limitations of single data sources. Twenty-six states to date have statewide 
longitudinal data systems to facilitate policy decision-making or accountability reporting.42 Many 
states have plans to expand and improve their system’s capacity to collect, link, and share 
longitudinal data within and across states. States have also developed standard codes across 
educational and labor market outcomes which enable tracking, comparison, and measurement of 
state educational and skill development progress. Combining these data sources would allow for 
identification of longitudinal trends in student outcomes, examination of variation across student 
demographics and other characteristics, and analysis of the links between education, the state 
economy, and workforce development. Researchers studying the impact of OBF are encouraged to 


41 Hillman, N. W., Fryar, A. H., and Crespin-Trujillo, V. (2017). Evaluating the impacts of performance funding in Ohio and Tennessee. 
American Educational Research Journal, 55(1): 1144-170. 

42 Armstrong, J. and Zaback, K. (2016). Assessing and Improving State Postsecondary Data Systems. Retrieved from 
http://www.ihep.org/sites/default/files/ uploads/postsecdata/docs/resources/state_postsecondary_data_systems.pdf 
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integrate these data sources in order to capture the variation in policies and their effect on different 
groups of students. 


Conclusion 


Arriving at an accurate and complete appraisal of state OBF policy is a challenging exercise. OBF 
policy is complex, changes frequently, and varies significantly by state. The same is true for 
important elements of the policy that are not formally codified, such as implementation timelines 
and strategies. For these reasons, complete and accurate descriptions of OBF policies requires in- 
depth examination of written policy, determination of the status of policy implementation, and 
member-checks with state policymakers. This exercise is foundational to arriving at an accurate 
understanding of how to categorize states when conducting analyses of the effects of OBF policies. 


States have taken varied and complex approaches to determining whether, which, and how 
certificates are rewarded by their OBF formulas. Moreover, certificates themselves vary 
substantially. To date, this nuance has not been reflected in the preeminent literature on OBF. Yet it 
is incumbent upon researchers to capture these variations as they assess the effects of OBF on 
certificate completion and debate whether and under what circumstances certificates should be 
included in OBF policy. This brief identifies a range of challenges as well as possible solutions to 
better capture important variations in certificates and how they are recognized in OBF formulas. 
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Appendix A: Sources for Table 1 


Arkansas: retrieved from 

http: //www.arkleg.state.ar.us/assembly/2011/2011R/Acts/Act1203.pdf 

Colorado: retrieved from https://highered.colorado.gov/Publications/General/1319/default.htm] 
Florida: retrieved from http://www.flbog.edu/board /office/budget/performance_funding.php 
Hawaii: retrieved from 

http://www.hawaii.edu/offices /app/hgi/A Performance Funding Model Description FY2019v2. 
df 

Illinois: retrieved from http: //legacy.ibhe.org/PerformanceFunding/PDF/Overview.pdf 
Indiana: retrieved from 

https://www.in.gov/che/files/2018 PF Evolution Notes 5 31 18 wnotes.pdf 

Kentucky: retrieved from http://cpe.ky.gov/policies/data/technicalguide.pdf 

Louisiana: retrieved from https://regents.la.gov/wp-content/uploads/2018/03/4c-Outcomes- 


Based-Funding-Formula-Overview.pdf 
Michigan: retrieved from 


http://www.mcca.org/uploads/ckeditor/files/Analysis%20as%20Passed%20By%20The%20Hous 
e%20(4 27 2016).pdf 


Montana: retrieved from http://mus.edu/data/performancefunding/MUS- 
PerformanceFundingCriteria5-19-16.pdf 


Nevada: retrieved from https://nshe.nevada.edu/wp-content/uploads/Funding-Model-Summary- 
Revised 8 12.pdf 

New Mexico: retrieved from 

http://www. hed.state.nm.us/uploads/files/NM%201%20%26%20G%20funding%20formula/New 
%20Mexico%20HED%20Funding%20Formula%20- 


%202017%20Technical%20Guide%20for%20FY18%20Budget%20Cycle.pdf 
New York: retrieved from https://www.suny.edu/about/leadership/board-of- 


trustees /meetings /webcastdocs/6%20- 
%20Job%20Linkage%20Incentive%20Funding%20Program%20Summary.pdf 


North Carolina: retrieved from https://www.nccommunitycolleges.edu/sites/default/files/state- 
board/finance/fc_12 - fy2017-18 state aid allocation and budget policies revised _08.16.2017.pdf 


Ohio: retrieved from https://www.ohiohighered.org/press/new-performance-based-model- 
higher-education-ohio 

Tennessee: retrieved from 
https://www.tn.gov/content/dam/tn/thec/bureau/fiscal_admin/fiscal_pol/obff/Detailed_Outcom 
es_Formula_Definitions_01-2016.pdf 

Texas: retrieved from https://tacc.org/sites/default/files/documents/2018- 

08/performance based funding for community colleges.pdf 

Utah: retrieved from https://higheredutah.org/higher-ed-appropriations-approves-performance- 
funding-model/ 

Virginia: retrieved from http://trcenter.vccs.edu/wp-content/uploads/2016/12/Performance- 


Funding-Measures-and-Definitions.pdf 

Washington: retrieved from https://www.sbctc.edu/about/agency/initiatives-projects/student- 
achievement-initiative.aspx 

Wisconsin: retrieved from 
http://www.wtcsystem.edu/wtcsexternal/cmspages/getdocumentfile.aspx?nodeguid=1d030934d- 
ecfe-4317-83af-4e3e3c2dbcad 
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Appendix B: Methodological Notes for Example Analyses 


This Appendix provides short descriptions for the analyses presented in this brief that use data 
from SLDS and/or workforce sources, explaining data definitions and our approach. 


Analysis in Table 4. Reported Number of Certificates Awarded in Tennessee by 
Data Systems (2016) 


This analysis is conducted to understand the discrepancy in the total number of short-term and 
long-term certificates earned if different credit hour requirements apply to the count. To align with 
Tennessee’s OBF policy, we applied the same definitions as Tennessee outlines in a comprehensive 
operational definition document of their OBF policy.43 We generated a total number of certificates 
earned by Tennessee community college students in academic year 2016 by short-term and long- 
term programs as defined by Tennessee’s credit hour requirements (less than 24 credit hours vs. 
24 credit hours or more). To compare with IPEDS data, we downloaded the 2016 completion data 


and examined the list of community colleges to make sure our institution sample is accurate and 
completely aligned. IPEDS completion data includes a variable named “Award Level,” which 
categorizes certificates into the classification in Table 3. We compared the number of certificates by 
different credit hour requirements in IPEDS and SLDS and present our findings in Table 4. 


The following table contains our data restrictors to Tennessee SLDS data according to their OBF 


policy metrics: 


Table B1. Data restrictors used in Tennessee SLDS data summary 


CERTIFICATE TYPE SLDS IPEDS 
Short-term Long-term Short-term Long-term 
certificates certificates certificates certificates 
Year 2016 2016 2016 2016 
Institution Level 2-yr colleges 2-yr colleges 2-yr colleges 2-yr colleges 
Credit hours (<24) Credit hours | Award level (less Award level 
Credit Hour Requirement (>=24) than one year) (one to two 
years) 
Excluding General — Excluding Excluding General Excluding 
Education General Education General 
certificates Education certificates Education 
associated with certificates associated with certificates 
Types of Certificates CIP code: associated CIP code: 24.0101 associated 
24.0101 & with CIP & 51.0000 with CIP code: 
51.0000 code: 24.0101 & 
24.0101 & 51.0000 
51.0000 
Citizenship Citizen only Citizen only Citizen only Citizen only 


43 Outcomes Based Funding Resources Tennessee Higher Education Commission and Student Assistant Corporation. Retrieved from 
https://www.tn.gov/thec/bureaus/finance-and-administration/fiscal-policy/redirect-fiscal-policy/outcomes-based-funding-formula- 
resources/redirect-outcomes-based-funding-formula-resources/2015-20-outcomes-based-funding-formula.html. 
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Analysis in Table 5. Reported Number of Certificates Awarded in Indiana by 
Data Systems (2016) 


This analysis is conducted to understand the discrepancy in the total number of long-term 
certificates earned if no residency status is applied to the count. According to Indiana’s policy, only 
the credentials earned by Hoosiers are considered in the outcomes.** We generated a total number 
of long-term certificates earned by students’ residency status in academic year 2016 and compared 
it with IPEDS completion data. IPEDS completion data does not account for state residency, only 
providing the total number of certificates earned by citizen and non-citizen students. The following 
table contains our data restrictors to Indiana SLDS data according to their OBF policy metrics: 


Table B2. Data restrictors used in Indiana SLDS data summary 


CERTIFICATE TYPE | SLDS | IPEDS 
Long-term certificates Long-term certificates 
Year 2016 2016 
Institution Level Two-year colleges Two-year colleges 
Residency/Citizenship Resident only Citizen only 


Analysis of Figure 3. Certificate production in Louisiana (Data Source: IPEDS 
2010-2016) 


We conducted this trend analysis to examine if there is a difference in changes over time between 
prioritized certificates and other certificates in Louisiana before its OBF was updated in FY 2017. 
Louisiana’s previous OBF requires certificates to be on the list of four- and five- star jobs defined by 
Louisiana Workforce Commission to be rewarded as outcomes. We downloaded workforce 
information from the Commission’s website, which provides a Standard Occupational Classification 
(SOC) code for each star job. To link the star jobs with corresponding major fields as used in IPEDS 
to report the number of certificates earned by programs, we converted the SOC codes to CIP codes 
by the NCES SOC to CIP Crosswalk. By linking the star job list with institutions’ completion data, we 
explored the change of certificate production among four- and five- star jobs and others in 
Louisiana before and after the implementation of OBF. 


Analysis of Wisconsin Top 50 Occupations (2016) 


Wisconsin’s policy requires certificates to be on the “Hot Job” list as determined by Wisconsin 
Department of Workforce Development.* Hot Jobs are “high projected growth occupations” with 
median salaries above the state median, for which the percent of growth is greater than state 
average, and in which there are the most projected job opportunities.46 We obtained the 2016 Hot 
Job list and matched the Standard Occupational Classification (SOC) codes for the Top 50 


44 Performance-Based Funding: FAQ. Indiana Commission for Higher Education. Retrieved from 
https://www.in.gov/che/files/Performance%20Funding%20FAQ%20FINAL. pdf 

45 Retrieved from https://www.jobcenterofwisconsin.com/wisconomy/pub/hotjobs 

46 Ibid. 
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occupations by their description. Occupations were only included if descriptions included 
certificates as their minimal education attainment requirement. Using the NCES SOC to CIP 
Crosswalk*7 to convert SOC code to CIP codes and linking to IPEDS completions data, we estimated 
the percent of students who earned certificates in these matched Hot Job fields as of the total 
number of certificates awarded in 2016. One limitation of this analysis is that we were not able to 
retrieve the Hot Job list prior to Wisconsin’s implementation of OBF in 2015, so this analysis is only 
descriptive of the proportion of certificates earned in Hot Job fields in 2016. 


47 Retrieved from https://nces.ed.gov/ipeds/cipcode/resources.aspx?y=55 
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